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Abstract. We prove that if a homeomorphism of a closed orientable surface 
S has no wandering points and leaves invariant a compact, connected set K 
which contains no periodic points, then either A" = S = T^, or iC is the 
intersection of a decreasing sequence of annuli. A version for non-orientable 
surfaces is given. 



I. Introduction. By aperiodic invariant continuum we mean a compact con- 
nected set which is invariant by some homeomorphism of a compact surface, and 
which contains no periodic points. We are interested in describing aperiodic in- 
variant continua of non- wandering homeomorphisms. This type of sets appear fre- 
quently when studying generic area-preserving diffeomorphisms, due to a resuh of 
Mather [MatSlj . which states that for such diffeomorphisms, the boundary of cer- 
tain open invariant sets (see Definition 12. ip is a finite union of aperiodic continua. 
Thus, having good topological information about aperiodic invariant continua is 
helpful to describe the dynamics of a C-generic area-preserving diffeomorphism. 
An example of this is the work of Franks and Le Calvez in |FL03| in the case that 
the surface is a sphere. 

Our main result is the following 

Theorem 1.1. Let f : S S be a homeomorphism of a compact orientable surface 
such that r2(/) ~ S. If K is an f -invariant continuum, then one of the following 
holds: 

(1) / has a periodic point in K ; 

(2) K is annular; 

(3) K = S^T^; 

By annular continuum we mean an intersection of a nested sequence of topolog- 
ical annuli (see Definition 12. 6p . When S is non-orientable, a version of Theorem 

II. 11 holds, however with two extra cases: K could be a non-separating continuum 
in a Mobius strip, and in the case that K ^ S, the surface could be or the Klein 
bottle (Corollary El]). 

An important result of [FLOSj states that for a generic area-preserving diffeomor- 
phism of the sphere, the stable and unstable manifolds of hyperbolic periodic points 
are dense. This fact was generalized to an arbitrary surface by Xia [Xia06| , and one 
of the main steps of his proof is obtaining a version of Theorem 11.11 which assumes 
generic conditions on the (area-preserving) diffeomorphism and is restricted to con- 
tinua which are the closure of a particular kind of open sets. Thus Theorem 11.11 
extends Xia's result to general homeomorphisms without wandering points (which 
includes area-preserving homeomorphisms), with no additional hypothesis on the 
continuum and no genericity conditions. 
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A question that motivates studying aperiodic invariant continua is the foUowing: 

Question 1.2. What are the possible obstructions to the transitivity of a -generic 
area-preserving dijfeomorphism? 

Bonatti and Crovisier proved in [BC04] that a C^-generic area-preserving dif- 
feomorphism of a compact manifold (of any dimension) is transitive. However, in 
dimension 2, it is known that this is not true in the C topology if r is large enough, 
because of the KAM phenomenon: There are open sets of diffeomorphisms where 
a C-generic element has an elliptic periodic point surrounded by invariant circles 
(see, for instance, jDou92| ). and this is an obstruction to transitivity. Hence the 
question is: is this the only possible obstruction? In other words, does the non- 
transitivity of a C-generic area-preserving diffeomorphism imply the existence of 
elliptic periodic points? 

Studying Question II. 2i aperiodic invariant continua appear naturally as bound- 
aries of invariant open sets. Theorem 11.11 implies that the presence of annular 
periodic continua is a necessary condition for the non-transitivity of a generic area- 
preserving diffeomorphism. In fact, a consequence of Theorem ll.ll is that the familly 
of aperiodic invariant continua which are minimal with respect to the property of 
being annular is pairwise disjoint (we call these continua frontiers, see |KN) for de- 
tails). This allows a sort of decomposition of the dynamics in terms of the aperiodic 
invariant continua. Similar concepts appear in the work of Jager IJ09] (where the 
word circloid is used instead of frontier) when studying nonwandcring homeomor- 
phisms of the torus with bounded mean motion. 

In [KNj , these observations play a fundamental role in the proof of the following 
result: for any r > 1, given a C-generic pair of area-preserving diffeomorphisms of 
a compact surface, the iterated function system (or, equivalently, the action of the 
semi-group) generated by them is transitive. 

We should mention that the basic idea of the proof of Theorem 11.11 is inspired 
by the analogous result from |FL03| in the case where the surface is a sphere. 

This article is organized as follows. In §1-5 we recall some background and 
results about ideal boundary points, continua, Lefschetz numbers and indices and 
we prove some elementary facts; in §5 we prove our main theorem, and a corollary 
about rotation numbers is mentioned; in §6 we state a version of the theorem for 
non-orientable surfaces, with an outline of the proof. 

Acknowledgments. I am grateful to M. Nassiri for motivating this problem, as 
well as L. N. Carvalho, J. Franks and E. R. Pujals for useful discussions. 

2. Ideal boundary, continua, and complementary domains. If [/ is a non- 
compact surface, a boundary representative of C/ is a sequence Pi Z) P2 ■ ■ ■ of 
connected unbounded (i.e. not relatively compact) open sets in U such that du Pn 
is compact for each n and for any compact set K C U, there is no > such that 
P„ n if = if n > TiQ. Two boundary representatives {Pi} and {P/} are said to 
be equivalent if for any n > there is m > such that P„i C P^, and vice- versa. 
The ideal boundary of U is defined as the set bi U of all equivalence classes of 
boundary representatives. We denote by U* the space f7 U bi J7 with the topology 
generated by sets of the form V UV, where V is an open set in U such that djj V 
is compact, and V denotes the set of elements of bi U which have some boundary 
representative {Pi} such that Pi C V for all i. We call U* the ideal completion of 
U. Any homeomorphism f : U ~> U extends to a homeomorphism f* : U* ^ U* 
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such that f*\u — f- If is orientable and hiU is finite, then U* is a compact 
orientable boundaryless surface. See [Ric63| and |AS60j for more details. 

From now on, S will denote a compact orientable surface. Let U be an open 
connected subset of S. For eachp* £ bj U, we write Z{p*) for the set cl5'(p|y VnU) 
where the intersection is taken over all neighborhoods V of p* in U* . It is easy to 
see that Z{p*) is a compact, connected, nonempty set (see |Mat82) ). 

Definition 2.1. We say that U d S is a complementary domain if it is a connected 
component of the complement of some compact connected subset of S. 

The next proposition is a direct consequence of |Mat81[ Lemma 2.3]. 

Proposition 2.2. IfU is a complementary domain in S, then it has finitely many 
ideal boundary points. 

If bi U is finite, for each p* E hiU we may choose a neighborhood V oi p such 
that V is homcomorphic to a closed disk, and such that V HhiU = {p}. Thus 
V \ {p} is a topological annulus in S. And, unless U is a topological disk, the 
boundary of V is an essential simple closed curve in S. From this, we have 

Proposition 2.3. If U is a complementary domain in S , then hi U is finite, and 
there is a compact bordered surface Su C U such that U \ Su has finitely many 
connected components, each of which is homcomorphic to an open annulus. 

Corollary 2.4. IfU is a complementary domain in S, then dU has finitely many 
connected components. 

Proof. Choose K C U such that U \ K is a finite union of disjoint annuli. If A is 
a connected component oi U \ K, then d A\U is connected (since it is Z{p*) for 
some p* € bi [/), and dU = IJ^ d A\U, where the union is taken over all connected 
components A oi U \K. Since these are finitely many, the claim follows. 

□ 

We remark that the number of boundary components of U may be smaller than 
the number of ideal boundary points, since the sets Z(p*), p* G hiU need not be 
disjoint. 

2.1. Continua. By a continuum we mean a compact connected set. 

Proposition 2.5. Let K be a continuum and U the family of all connected com- 
ponents of S \ K . Then all but finitely many elements of hi are simply connected. 

Proof. We consider two cases. First suppose that for some U GU, there is a simple 
closed curve 7 which is homotopically nontrivial in U but trivial in S. Let D be 
the topological disk bounded by 7 in S. Since 7 is nontrivial in U , there is some 
point of K in D. Since K is connected and j C S \ K, it follows that K C D. 
Thus if U' e U and U' ^ U, then U' C D. From this we conclude that U' is 
simply connected. Indeed, if 7' is a homotopically nontrivial simple closed curve 
in [/', then by a similar argument it bounds a disk D' G D which intersects K, so 
K C D'. But this implies that S \ D' C U' (because it is connected) so U' = U , a 
contradiction. Therefore, all but one element of U are simply connected. 

Now suppose that for every J7 G W, if 7 is homotopically nontrivial in U then 
it is also homotopically nontrivial in S, and assume that there are infinitely many 
complementary domains Ui,U2, . ■ . oi K which are not simply connected. For each 
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Ui, let 7i be a simple homotopically nontrivial simple closed curve in Ui. By our 
assumption, ji is also nontrivial in S. The curves {7^ : i £ N} are pairwise disjoint, 
so there must be infinitely many of them in the same homotopy class of S. But if, 
say, 71, 72 and 73 are all homotopic and disjoint, there are two disjoint annuli Ai 
and A2 such that (up to reordering the indices) d Ai = 71 U72 and d A2 = 72 U73. 
Since the boundary of Ai contains points of two different connected components 
oi S \ K, it is clear that Ai must intersect K. Since K is connected, it follows 
that K C Ai. But with the same argument we also conclude that K C A2, a 
contradiction. This completes the proof. □ 

2.2. Annular continua. 

Definition 2.6. A continuum K C S is said to be annular if it has a neighborhood 
A G S homeomorphic to an open annulus such that A\K has exactly two compo- 
nents, both homeomorphic to annuli. We call any such A an annular neighborhood 
of K. 

This definition is equivalent to saying that K is the intersection of a sequence 
{Ai} of closed topological annuli such that Ai^i is an essential subset of Ai (i.e. it 
separates the two boundary components of Ai), for each i G N. 

3. Indices and Lefschetz number. If / is a homeomorphism and D is a closed 
topological disk without fixed points in its boundary, we denote by Ind/(Z3) the 
fixed point index of / in D. (see jPolSOj ). If there are finitely many fixed points 
of / in D, then Ind/(Z?) is equal to the sum of the Lefschetz indices of these fixed 
points. If Di, . . . , Dn are disjoint disks such that the set of fixed points of / is 
contained in the interior of their union, then we have the Lefschetz formula: 

n 

^Ind/(A:)-L(/) 

i=l 

where L{f) denotes the Lefschetz number of /. 

Lemma 3.1. Let S be an orientable closed surface with Euler characteristic x{^) ^ 
0. Then, for any homeomorphism f : S —^ S there is n > such that the Lefschetz 
number of f" is non-positive: L{f^) < 0. 

Proof When x{S) < 0, a proof can be found in |Xia06| . If xiS) = 0, then 5 ~ T^, 
and the automorphism induced by / on Hi{S,Q) can be represented by a matrix 
A G SL(2,Z). It is well known that any such matrix is either periodic {A" = I for 
some n > 0, so tr(A") — 2), parabolic (and then tr(A^) = 2) or hyperbolic (and 
then tr(A2) > 2). In cither case, there is n such that L(/") = 2 - tr(A") < 0. □ 

4. Wandering points. Given a homeomorphism / ; S ^ S,we say that a nonempty 
open set U is wandering if f^{U) fl C/ = for all n > (or, equivalently, for all 
n ^ Q). We denote by ^{f) the set of non- wandering points of /. That is, the 
(compact, invariant) set of points which have no wandering neighborhood. 

Remark 4.1. We will use the following observations several times: 

(1) If Vl{f) = S, then r2(/") = S. To see this, given a nonempty open set Uq we 
can define recursively Ui+i — f'''+^{Ui) fl Ui where ki+i > is chosen such 
that the intersection is nonempty. Then there are integers ii < 12 < • • ■ < in 
such that fcij = ki^ = • • ■ = fci,^(modn), so that fci^ + • • ■ + fci„ — mn for 
some m > 0, and it is easy to verify that /™"(C/o) fl t/o ^ 0. 



APERIODIC INVARIANT CONTINUA FOR SURFACE HOMEOMORPHISMS 



5 



(2) If Q{f) — S and {Ui}i(zjq is a family of pairwise disjoint open sets which 
are permuted by / (e.g. the connected components of the complement of a 
compact periodic set) then each Ui is periodic for /. 

Lemma 4.2. Let D C S be a topological open disk and f:D^D a homeomor- 
phism. Suppose that there is a neighborhood of d D in D which does not contain the 
positive or the negative orbit of any wandering open set, and f has no fixed points 
in dD. Then the index of the set of fixed points of f in D is 1. In other words, 
there is a closed topological disk D' which contains all fixed points of f in D, such 
that lndf{D') = 1. 

Proof. Since it contains no fixed points, d D is not reduced to a single point. By a 
theorem of Cartwright and Littlewood [CL51| (see also |FL03I Proposition 2.1]), the 
extension / of /|d to the prime ends compactification D of D has no fixed points 
in the boundary circle dD. Thus / is orientation-preserving, and Ind^(-D) — 1, 

and since fixed points of / are in a compact subset of D, we can choose a closed 
disk D' C D containing all fixed points of /, so that Ind^-(£'') = Ind^-(L)) = 1. But 

since D' C D and flo' = flo', it follows that Indy(_D') = lndf{D') and we are 
done. □ 

5. Main theorem. We begin with a brief outline of the proof. The idea is to 
generalize the index argument used in |FL03| for the case of the sphere. However, 
to do that we need to modify the underlying manifold: we consider the (possibly 
infinitely many) connected components oi S \ K. The non-wandering hypothesis 
guarantees that these components are permuted by /. Since these components 
are complementary domains, they have finitely many ends. Next we "remove" 
every nontrivial component (except for a neighborhood of its boundary) , leaving a 
bordered submanifold N of S which is a neighborhood of K. We can modify f\^ 
obtaining a map which coincides with / in a neighborhood of K, but which leaves 
the boundary of N invariant. After collapsing the boundary circles of N to points, 
we obtain a new compact surface containing K, and a homeomorphism which has 
no periodic points on K, and by a Lefschetz index argument we conclude that this 
surface can only be a sphere. From this we conclude easily that K is annular. 

Remark 5.1. If K is an aperiodic invariant continuum and K ^ S, then Theorem ll.il 
implies that K is annular. Following ^FL03. §3] (using a small annular neighborhood 
A of K, and lifting / to the universal covering of A) one can define the rotation set 
Pf{K) C M (which is defined modulo integer translations). Now, with almost no 
modifications, the proof of [FL03i Proposition 5.2] remains valid. Thus we obtain 
the following 

Corollary 5.2. ///: S —^ S is an area preserving homeomorphism and K C. S is 
an invariant continuum with no periodic points, then K is annular, Pf{K) consists 
of a single irrational number a, and the rotation numbers in the prime ends from 
both .sides of K coincide (up to a .sign change) with a. 

5.1. Proof of Theorem \l.l\ We may assume that / is orientation-preserving (other- 
wise consider instead of /). If K = S and / has no periodic points, then 5* = 
by the Lefschetz theorem, and we are done. Now suppose that / has no periodic 
points in K and K ^ S. We need to show that K is annular. 
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Consider the family V of connected components of S\K which are not topological 
disks, which is finite by Proposition 12.51 Since open sets are nonwandering, each 
element of V is periodic by /. Choosing a power of / instead of / we may (and we 
do from now on) assume that each element of V is fixed by /. 

Since each e V is a complementary domain, by Proposition 12 . 31 we can choose 
a compact surface with boundary Sy C V such that V \ Sy has finitely many 
components, all of which are annuli. 

Given y G V, the ideal boundary points of V are periodic by so by taking 

a power of / instead of / we may assume that they are in fact fixed. This implies 
that if 7 is a sufficiently small closed loop in V which bounds a disk containing p* in 
V* , then /(7) is homotopic to 7 in y (and thus in S). Moreover, f{Z{p*)) = Z{p*) 
for any p* in bi V. Note also that Z{p*) C K for aU p* e bi V. 

Let Ai, . . . , An be the connected components of V \ Sv ■ Each Ai is a topological 
annulus, whose boundary in S is given by a loop 7^ and the continuum Zi = AiHK 
(which is Z{p*) for some p* e V*). Since f{Zi) = Zi, if ai C Ai is an essential 
simple closed curve close enough to Zi, we have that f{(7i) C Ai. Since f{(7i) is 
homotopic to ai in Ai, there exists a homeomorphism hi: Ai — > Ai which maps 
/(cTj) to ai and which is the identity in a neighborhood of the boundary of Ai; 
furthermore, we may assume that hi{x) — f^^{x) for x G /(^i) (see |Eps66| ) . 
Extending hi to the identity outside Ai, and letting / = /ii . . . /i„/, we get an 
orientation preserving homeomorphism such that f{x) = f{x) for x £ S \ VJiAi and 
/(ci) = ai. If Sv is the surface bounded by ai,. . . ,an which intersects Sy, we 
have that f[Sv) — Sv and / is the identity on the boundary of Sv- We do this for 
each y G V, and finally we consider the boundaryless compact surface S obtained 
by collapsing each boundary circle oi S\ 'mi\JvevSv to a point, and the induced 
homeomorphism which we still call /, for which these points are fixed (see figure 
[T]). This new surface contains S \ UV, and / coincides with / on that set. Each 
V & V was replaced by a (finite) union of one or more invariant topological disks, 
and the boundary of each of these disks is contained in K (and hence, it contains 
no periodic points). 



s 



K 




Figure 1 . The complement of K in S consists of topological disks 
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Since V consists of all components oi S \ K which are not disks, from our con- 
struction we see that all components oi S \ K are topological disks. 

Suppose that xi^) ^ 0- Then by Lemma [XT] there is n such that L{f^) < 0. 
Let D be a connected component oi S \ K such that f"{D) — D. We know that 
/" coincides with /" in a neighborhood oi d D C K, so the fact that /" has 
no wandering points (and no fixed points in K) implies that the hypotheses of 
Lemma |4?2] hold . Hence, the fixed point index of /" in D must be 1 (in particular, 
D contains a fixed point). From this, it follows that there are finitely many /"- 
invariant components in S\K. In fact, if there were infinitely many, then one could 
find a sequence of fixed points accumulating in K, which contradicts the aperiodicity 
of K. Moreover, we may assume that there is at least one such component (by 
starting with an appropriate power of / instead of /). 

Since /" has no fixed points in denoting the components oi S \ K which are 
/"-invariant by Di, . . . , _D/c, we have from the Lefschetz formula 

k 

1=1 

which contradicts our choice of n. From this we conclude that x{S) > 0, hence S 
is a sphere. 

But then, since / preserves orientation, L{f™') = x{S) = 2 for all m. This 
implies that S\K consists of exactly two components Di and Z?2- In fact, if there 
were more than two such components, it would be possible to choose m such that 
/™ leaves three or more of those components fixed, so that, repeating our previous 
argument, L(/'") > 3, contradicting our previous claim. 

Since Di and Z?2 are topological disks, each of them is the union of an increasing 
sequence of closed topological disks, so that K is the intersection of a decreasing 
sequence of annuli {A„}. These annuli are eventually contained in any neighbor- 
hood of K, which means that, for some no, {A„}„>„q is a decreasing sequence of 
annuli in the original surface S, and n„>„oA„ = K. Thus K is annular in S. This 
completes the proof. □ 

6. Non-orientable case of Theorem 11.11 

Corollary 6.1. Let f : S ^ S be a homeomorphism of the closed non-orientable 
surface S, such that VL{f) — S. If K is an f -invariant continuum, then one of the 
following holds: 

(1) / has a periodic point in K ; 

(2) K is annular; 

(3) K is the intersection of a nested sequence of Mobius strips; 

(4) K = S = Klein bottle. 

Proof. We consider the oriented double covering tt: S ^ S, and a lift f : S ^ S 
of /. Since / has no wandering points, that must be true of / as well. In fact, if 
U C S is a. sufficiently small open set, then tt~^{tt{U)) — UUU' where the union is 
disjoint and U' is homcomorphic to U. If n > is such that /"(7r(f7)) n tt{U) ^ 
then either P{U) n U ^ 9 or V' = P{U) r\(j' ^ %. If the latter case holds, 
then again 7r^^(7r(V^')) is the disjoint union of V' and V, where V dlJ, and there 
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is m > such that /"(f) n / (which imphes that /™+"(;7) n f7) ^ 0) or 
/™(y) r\V ^% (which imphes /"(JJ) n J7 ^ 0), so [/ is nonwandering. 

Now 7r~ ^(iiT) consists of either a unique connected /-invariant set or two copies of 
K which are invariant by / if the hft is chosen appropriately. Let K be one of those 
components (or the unique component if there is only one) . If K has no periodic 
points of /, then / cannot have a periodic point in if, because periodic points of / 
project to periodic points of /. Thus we are in the setting of Theorem I LI) and we 
conclude that cither K is annular or S' = T^. In the latter case, it follows that S 
is a Klein bottle. In the former case, we have a decreasing sequence of topological 
annuli {v4i}igN such that K = Ai. The sets Ai project to a decreasing sequence 
of neighborhoods {AijigN of if, each of which is either homeomorphic to an annulus 
(in which case it projects injectively) or to a Mobius strip, and it is easy to see that 
K — p|,- Ai. By taking a subsequence of {^i}igN if necessary, we see that either (2) 
or (3) must hold. □ 
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